What happened in the Reproducible Builds effort between Sunday November 6 and Saturday November 12 2016: Media coverage Matthew Garrett blogged about Tor, TPMs and service integrity attestation and how reproducible builds are the base for systems integrity. The Linux Foundation announced renewed funding for us as part of the Core Infrastructure Initiative. Thank you! Outreachy updates Maria Glukhova has been accepted into the Outreachy winter internship and will work with us the Debian reproducible builds team. To quote her words

siamezzze: I've been accepted to #outreachy winter internship - going to
work with Debian reproducible builds team. So excited about that! <3
Debian

Toolchain development and fixes dpkg:

• Thanks to a series of dpkg uploads by Guillem Jover, all our toolchain changes are now finally available in sid!
• This means your packages should now be reproducible without having to use our custom APT repository.
• Ximin Luo opened #843925 to remind the fact that dpkg-buildpackage should sign buildinfo files.
• We hope to have detailed post about the new dpkg and the new .buildinfo files for debian-devel-announce soon!

debrebuild:

• srebuild / debrebuild work was resumed by Johannes Schauer and others in #774415.

Bugs filed Chris Lamb:

• #844102 filed against lava-dispatcher
• #844103 filed against lava-server
• #844111 filed against python-defaults
• #843698 filed against tunnelx
• #843967 filed against asyncpg
• #843865 filed against keystone
• #844101 filed against suil

Daniel Shahaf:

• #844232 filed against daisy-player
• #844236 filed against libhtml-lint-perl
• #844228 filed against ebook-speaker

Niko Tyni:

• #843432 filed against libwww-curl-perl

Reiner Herrman:

• #844095 filed against fdm
• #844096 filed against gringo
• #843469 filed against dpkg

Reviews of unreproducible packages 136 package reviews have been added, 5 have been updated and 7 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated:

• Added: random_order_in_dh_pythonX_substvars, valac_permutes_get_type_calls
• Updated: timestamps_in_static_libraries

Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by:

• Chris Lamb (29)
• Niko Tyni (1)

diffoscope development A new version of diffoscope 62~bpo8+1 was uploaded to jessie-backports by Mattia Rizzolo. Meanwhile in git, Ximin Luo greatly improved speed by fixing a O(n²) lookup which was causing diffs of large packages such as GCC and glibc to take many more hours than was necessary. When this commit is released, we should hopefully see full diffs for such packages again. Currently we have 197 source packages which - when built - diffoscope fails to analyse. buildinfo.debian.net development

• Submissions with duplicate Installed-Build-Depends entries are rejected now that a bug in dpkg causing them has been fixed. Thanks to Guillem Jover.
• Add a new page for every (source, version) combination, for example diffoscope 62.
• DigitalOcean have generously offered to sponsor the hardware buildinfo.debian.net is running on.

tests.reproducible-builds.org Debian:

• For privacy reasons, the new dpkg-genbuildinfo includes Build-Path only if it is under /build. HW42 updated our jobs so this is the case for our builds too, so you can see the build path in the .buildinfo files.
• HW42 also updated our jobs to vary the basename of the source extraction directory. This detects packages that incorrectly assume a $pkg-$version directory naming scheme (which is what dpkg-source -x gives but is not mandated by Debian nor always-true) or that they're being built from a SCM.
• The new dpkg-genbuildinfo also records a sanitised Environment. This is different in our builds, so HW42, Reiner and Holger updated our jobs to hide these differences from diffoscope output.
• Package-set improvements:
  • Holger refactored the create_meta_pkg_sets job so that it's easier to add new package sets.
  • This job is now also using dose-extra from jessie-backports so that it can deal with versioned provides.
  • Added 4 new package sets: debian-edu, debian-edu_build-depends, maint_pkg-grass-devel, maint_debian-accessibility, maint_pkg-openstack.
  • Switched to using the new URL for tails manifests to generate the tails package set.
  • Renamed maint_lua to maint_debian-lua
• Valerie Young contributed four patches for our long-planned transition from SQLite to PostgreSQL.
• In anticipation of the freeze, already-tested packages from unstable and testing on amd64 are now scheduled with equal priority.

reproducible-builds.org website F-Droid was finally added to our list of partner projects. (This was an oversight and they had already been working with us for some time.) Misc. This week's edition was written by Ximin Luo and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

It all started last June when my son had his Janoi (Yagnopavita) ceremony -- the ritual by which a Brahmana boy becomes "twice-born" and eligible to study the Vedas. As well as a profound religious experience, it is also an important social occasion with a reception for as many friends and family as can attend. (I think our final guest total was ~250.) This meant new outfits for everyone which might be exciting for some people but not me. I still don't know why I couldn't just keep wearing the khes and pitambar from the puja but no, apparently that's a faux pas. So I relented and agreed to wear my "darbari" suit from my wedding. And it didn't fit. I knew I had gained some weight in the intermediate 17 years but the thing was sitcom levels of too small. I ended up having to purchase a new one, a snazzy (and shiny!) maroon number with gold stripes (or were they black stripes?) Problem having been solved, much was eaten, more weight was gained and then I forgot about the whole thing. Tip 1: Actually Do Something. I have over the years tried to improve my physical condition but it has never gotten very far. For instance I have a treadmill/coatrack and a couple of years ago I began using it in earnest. I got to the point where I actually ran a 10K race without dying. But I did not train systematically and I ended up in some pain which caused me to stop working out for a while and then I never got around to restarting. Diets have also failed because I don't have a clear idea of what and how much I am eating. All I know is that women go into the kitchen and when they come out they have food. By what eldritch process this occurs is a mystery, I just eat whats given to me thankful that the magic happens. Once I was moved to try and help but quickly fell afoul of the lack of well-defined algorithms in Gujarati home cooking.

"How much saffron should I add?"

"this much."

"How much is this much in SI units?"

"You're annoying me. Get out."

Fast forward to March of this year. For my birthday, my wife got me a Fitbit fitness tracker. This is what I had needed all this time. It measure heart rate, distance travelled, time slept and several other pieces of info you can use to really plan a fitness regimen rationally. For example, I was chagrined to learn that sometimes when I'm at the computer, I am so immobile that the fitbit thought I was asleep. So I started planning to taken more frequent breaks. (A recent firmware upgrade has added the ability to nudge to walk atleast 250 paces each daytime hour which is handy for this.) Also by checking my heart rate I discovered that I went on the treadmill I ran too fast thereby stressing my body for little gain and ending up going too slow to get much aerobic effect. Now I can pace myself appropriately for maximum cardiac efficiency without ending up injuring myself and giving up. I also get a little more activity each day by simple changes such as taking the stairs instead of the lift and instead of getting off at the 14th street PATH I go all the way to 34th street and walk down. Tip 2: You must have data in order to see what you did right or wrong and to plan what you need to do moving forward. One caveat about these fitness trackers. They are not anywhere as accurate as a proper checkup from a doctor who specializes in such things. If you want to do any kind of pro or amateur athletics you probably should not rely on them but for the average shlub who just wants to avoid appearing on the news being winched off his sofa by the fire brigade they are good enough. Another practice I began was keeping a food diary. It can be a real eye-opener to see how much you are actually eating. It is probably much more than you thought. I am fortunate that my diet is pretty good to begin with. Vegetarian, (not vegan, Hindus eat dairy products,) mostly home-cooked with fresh ingredients, not fried or processed, and I don't drink alcohol. However there were a few optimizations I could make. I drink a lot of soda; atleast two cans a day. I really ought to stop altogether but in lieu of that I have atleast switched from Coke to Coke Zero thereby saving a lot of empty calories. I now eat 4 rotlis with my dinner instead of six. We as a family eat more green vegetables instead of potatos, skim milk instead of whole fat, canola oil instead of corn oil, and less rice and don't slather ghee on everything quite so much. One entirely new practice I've adopted that may seem faddish but works for me is intermittent fasting. The idea is to steadily train your body to need less food by eating all your days allowed amount pf calories during a 6-8 hour window and not eating at all during the remaining time. It's hard to get used to for many people but I fast atleast 2-3 times a month for religious reasons anyway so I adapted pretty quickly. The fitbit tells me how many calories I am expending and how many I can eat to maintain a healthy level of weight loss but other than that I don't bother with "food groups" or specific diets such as paleo, or low-carb etc. As long as what you eat is reasonably balanced and you are burning more calories than you are adding, it should be enough for weight loss. Indeed from the end of March to now, I've lost 3 stones (20Kg) even with the occasional "cheat" day. Tip 3: All published diets are bullshit without scientifically proven efficacy. Don't bother with them. Experiment instead and see what works for you and your metabolism. As long as you are getting all the proper nutrients (you shouldn't need a supplement unless you have an actual medical condition.) and you have a net calorie deficit, it's all good. If you eat food you enjoy, you are more likely to stick to your diet. The proper amount of sleep is one area of a healthy lifestyle I am still doing poorly in and the reasons are not all raven-related. I have always had problems with insomnia and was once actually diagnosed with sleep apnea. Losing weight has helped a lot but the fitbit is still reporting that I toss and turn a lot during the night. And that's when I'm in bed in the first place. I stay up much too late which can also lead to subsidiary bad behaviours such as midnight snacking. It's something I need to work on. Tip 4: Stop blogging at all hours of the night, It's not doing you any good. So that's what I'm doing. Moving forward, I need to deal with the sleep thing and I would also like to start some program of strength-training, I'm doing ok in terms of aerobic exercise but from what I've read, you also have to build up muscles to keep weight loss permanent. The difficulty is that it would involve joining a gym and then actually going to that gym so I've put it off for now. The immediate threat is Diwali (and Thanksgiving and Christmas...) My wife bought 4 lbs of sweets today and I can feel their presence in the fridge calling to me.

Aaargh gcc 5.x You Suck I had to write a quick program today which is going to be run many thousands of times a day so it has to run fast. I decided to do it in c++ instead of the usual perl or javascript because it seemed appropriate and I've been playing around a lot with c++ lately trying to update my knowledge of its' modern features. So 200 LOC later I was almost done so I ran the program through valgrind a good habit I've been trying to instill. That's when I got a reminder of why I avoid c++.

==37698== HEAP SUMMARY:
==37698==     in use at exit: 72,704 bytes in 1 blocks
==37698==   total heap usage: 5 allocs, 4 frees, 84,655 bytes allocated
==37698== 
==37698== LEAK SUMMARY:
==37698==    definitely lost: 0 bytes in 0 blocks
==37698==    indirectly lost: 0 bytes in 0 blocks
==37698==      possibly lost: 0 bytes in 0 blocks
==37698==    still reachable: 72,704 bytes in 1 blocks
==37698==         suppressed: 0 bytes in 0 blocks

One of things I've learnt which I've been trying to apply more rigorously is to avoid manual memory management (news/deletes.) as much as possible in favor of modern c++ features such as std::unique_ptr etc. By my estimation there should only be three places in my code where memory is allocated and none of them should leak. Where do the others come from? And why is there a missing free (or delete.) Now the good news is that valgrind is saying that the memory is not technically leaking. It is still reachable at exit but that's ok because the OS will reclaim it. But this program will run a lot and I think it could still lead to problems over time such as memory fragmentation so I wanted to understand what was going on. Not to mention the bad aesthetics of it. My first assumption (one which has served me well over the years) was to assume that I had screwed up somewhere. Or perhaps it could some behind the scenes compiler magic. It turned out to be the latter -- sort of as I found out only after two hours of jiggling code in different ways and googling for clues. That's when I found this Stack Overflow question which suggests that it is either a valgrind or compiler bug. The answer specifically mentions gcc 5.1. I was using Ubuntu LTS which has gcc 5.4 so I have just gone ahead and assumed all 5.x versions of gcc have this problem. Sure enough, compiling the same program on Debian stable which has gcc 4.9 gave this...

==6045== 
==6045== HEAP SUMMARY:
==6045==     in use at exit: 0 bytes in 0 blocks
==6045==   total heap usage: 3 allocs, 3 frees, 10,967 bytes allocated
==6045== 
==6045== All heap blocks were freed -- no leaks are possible
==6045== 

...Much better. The executable was substantially smaller too. The time was not a total loss however. I learned that valgrind is pronounced val-grinned (it's from Norse mythology.) not val-grind as I had thought. So I have that going for me which is nice.

Once I got CI working on multiple platforms the obvious next step was to be able to aggregate coverage reports across them. This should not be that hard, right? Well I've spent couple of hours on that during last few days. On Linux and OSX it was pretty much straightforward. Both GCC and Clang do support coverage, so it's just matter of configuring them properly and collect the coverage reports. I've used own solution for that in past and that was really far from working well (somehow I never managed to get coverage fully uploaded to Codecov). Fortunately there exists CMake script called CMake-codecov which does all needed work and works out of the box on GCC and Clang (even on OSX). Well it works on Travis only once you update the compilers and install llvm-cov tool. The Windows part on AppVeyor was much harder for me. This can be heavily accounted to lack of my experience with Windows and especially development on Windows in past ten years (probably even more). First challenge was to find something what can generate code coverage there. After lot of googling I've settled down on OpenCppCoverage what seems to be the only free solution I was able to find. The good thing is that it can generate coverage in Cobertura format that Codecov undestands. There are also bad things that I've learned. First of all it's quite hard to integrate this with CTest. There is no support for wrapping test calls in custom commands, so I've misused the memory checks for that purpose. I've written small python script which pretends the valgrind interface and does call OpenCppCoverage in the background. Now I had around 800 coverage files (one for each test case) and we need to deal with them somehow. The Codeconv command line client doesn't deal wit this out of the box so the obvious choice was to merge them before upload. There even seems to be script doing that, but unfortunately trying that on our coverage data make it nowhere near completion within hour, so that's not really good choice. Second thing I've tried was merging binary coverage in OpenCppCoverage and then exporting to Cobertura format. Obviously Gammu is special project as all I got from this attempt was crashing OpenCppCoverage (it did merge some of the coverages, but it failed in the end without indicating any error). In the end I've settled down to uploading files in chunks to Codecov. This seems to work quite okay, though is a bit slow, mostly due to way how Codecov bash uploader prepares data to upload (but this will be hopefully fixed soon). Anyway the goal has been reached, both Windows and Linux code shows in coverage reports.

Filed under: Debian English Gammu 0 comments

My descent began with a user reporting a bug and I fear I am still on my way down.

The bug was simple enough, a windows bitmap file caused NetSurf to crash. Pretty quickly this was tracked down to the libnsbmp library attempting to decode the file. As to why we have a heavily used library for bitmaps? I am afraid they are part of every icon file and many websites still have favicons using that format.

Some time with a hex editor and the file format specification soon showed that the image in question was malformed and had a bad offset header entry. So I was faced with two issues, firstly that the decoder crashed when presented with badly encoded data and secondly that it failed to deal with incorrect header data.

This is typical of bug reports from real users, the obvious issues have already been encountered by the developers and unit tests formed to prevent them, what remains is harder to produce. After a debugging session with Valgrind and electric fence I discovered the crash was actually caused by running off the front of an allocated block due to an incorrect bounds check. Fixing the bounds check was simple enough as was working round the bad header value and after adding a unit test for the issue I almost moved on.

Almost...

We already used the bitmap test suite of images to check the library decode which was giving us a good 75% or so line coverage (I long ago added coverage testing to our CI system) but I wondered if there was a test set that might increase the coverage and perhaps exercise some more of the bounds checking code. A bit of searching turned up the american fuzzy lop (AFL) projects synthetic corpora of bmp and ico images.

After checking with the AFL authors that the images were usable in our project I added them to our test corpus and discovered a whole heap of trouble. After fixing more bounds checks and signed issues I finally had a library I was pretty sure was solid with over 85% test coverage.

Then I had the idea of actually running AFL on the library. I had been avoiding this because my previous experimentation with other fuzzing utilities had been utter frustration and very poor return on investment of time. Following the quick start guide looked straightforward enough so I thought I would spend a short amount of time and maybe I would learn a useful tool.

I downloaded the AFL source and built it with a simple make which was an encouraging start. The library was compiled in debug mode with AFL instrumentation simply by changing the compiler and linker environment variables.

$ LD=afl-gcc CC=afl-gcc AFL_HARDEN=1 make VARIANT=debug test
afl-cc 2.32b by <lcamtuf@google.com>
afl-cc 2.32b by <lcamtuf@google.com>
 COMPILE: src/libnsbmp.c
afl-cc 2.32b by <lcamtuf@google.com>
afl-as 2.32b by <lcamtuf@google.com>
[+] Instrumented 751 locations (64-bit, hardened mode, ratio 100%).
      AR: build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/libnsbmp.a
 COMPILE: test/decode_bmp.c
afl-cc 2.32b by <lcamtuf@google.com>
afl-as 2.32b by <lcamtuf@google.com>
[+] Instrumented 52 locations (64-bit, hardened mode, ratio 100%).
    LINK: build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_bmp
afl-cc 2.32b by <lcamtuf@google.com>
 COMPILE: test/decode_ico.c
afl-cc 2.32b by <lcamtuf@google.com>
afl-as 2.32b by <lcamtuf@google.com>
[+] Instrumented 65 locations (64-bit, hardened mode, ratio 100%).
    LINK: build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_ico
afl-cc 2.32b by <lcamtuf@google.com>
Test bitmap decode
Tests:606 Pass:606 Error:0
Test icon decode
Tests:392 Pass:392 Error:0
    TEST: Testing complete

I stuffed the AFL build directory on the end of my PATH, created a directory for the output and ran afl-fuzz

afl-fuzz -i test/bmp -o findings_dir -- ./build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_bmp @@ /dev/null

The result was immediate and not a little worrying, within seconds there were crashes and lots of them! Over the next couple of hours I watched as the unique crash total climbed into the triple digits.

I was forced to abort the run at this point as, despite clear warnings in the AFL documentation of the demands of the tool, my laptop was clearly not cut out to do this kind of work and had become distressingly hot.

AFL has a visualisation tool so you can see what kind of progress it is making which produced a graph that showed just how fast it managed to produce crashes and how much the return plateaus after just a few cycles. Although it was finding a new unique crash every ten minutes or so when aborted.

I dove in to analyse the crashes and it immediately became obvious the main issue was caused when the test tool attempted allocations of absurdly large bitmaps. The browser itself uses a heuristic to determine the maximum image size based on used memory and several other values. I simply applied an upper bound of 48 megabytes per decoded image which fits easily within the fuzzers default heap limit of 50 megabytes.

The main source of "hangs" also came from large allocations so once the test was fixed afl-fuzz was re-run with a timeout parameter set to 100ms. This time after several minutes no crashes and only a single hang were found which came as a great relief, at which point my laptop had a hard shutdown due to thermal event!

Once the laptop cooled down I spooled up a more appropriate system to perform this kind of work a 24way 2.1GHz Xeon system. A Debian Jessie guest vm with 20 processors and 20 gigabytes of memory was created and the build replicated and instrumented.

To fully utilise this system the next test run would utilise AFL in parallel mode. In this mode there is a single "master" running all the deterministic checks and many "secondary" instances performing random tweaks.

If I have one tiny annoyance with AFL, it is that breeding and feeding a herd of rabbits by hand is annoying and something I would like to see a convenience utility for.

The warren was left overnight with 19 instances and by morning had generated crashes again. This time though the crashes actually appeared to be real failures.

$ afl-whatsup sync_dir/
Summary stats
=============

       Fuzzers alive : 19
      Total run time : 5 days, 12 hours
         Total execs : 214 million
    Cumulative speed : 8317 execs/sec
       Pending paths : 0 faves, 542 total
  Pending per fuzzer : 0 faves, 28 total (on average)
       Crashes found : 554 locally unique

All the crashing test cases are available and a simple file command immediately showed that all the crashing test files had one thing in common the height of the image was -2147483648 This seemingly odd number is actually meaningful to a programmer, it is the largest negative number which can be stored in a 32bit integer (INT32_MIN) I immediately examined the source code that processes the height in the image header.

if ((width <= 0)   (height == 0))          
        return BMP_DATA_ERROR;
if (height < 0)      
        bmp->reversed = true;   
        height = -height;        
 

The bug is where the height is made a positive number and results in height being set to 0 after the existing check for zero and results in a crash later in execution. A simple fix was applied and test case added removing the crash and any possible future failure due to this.

Another AFL run has been started and after a few hours has yet to find a crash or non false positive hang so it looks like if there are any more crashes to find they are much harder to uncover.

Main lessons learned are:

• AFL is an easy to use and immensely powerful and effective tool. State of the art has taken a massive step forward.
• The test harness is part of the test! make sure it does not behave in a poor manner and cause issues itself.
• Even a library with extensive test coverage and real world users can benefit from this technique. But it remains to be seen how quickly the rate of return will reduce after the initial fixes.
• Use the right tool for the job! Ensure you head the warnings in the manual as AFL uses a lot of resources including CPU, disc and memory.

not much to report but I got at least some RC bugs fixed in the last weeks. again mostly perl stuff:

• #759979 src:simba: "simba: FTBFS: RoPkg::Rsync ...failed! (needed)"
  keep ExtUtils::AutoInstall from downlaoding stuff, upload to DELAYED/7
• #817549 src:libropkg-perl: "libropkg-perl: Removal of debhelper compat 4"
  use debhelper compatibility level 5, upload to DELAYED/7
• #832599 iodine: "Fails to start after upgrade"
  update service file and use deb-systemd-helper in postinst
• #832832 src:perlbrew: "perlbrew: FTBFS: Tests failures"
  add patch to deal with removed old perl version (pkg-perl)
• #832833 src:libtest-valgrind-perl: "libtest-valgrind-perl: FTBFS: Tests failures"
  upload new upstream release (pkg-perl)
• #832853 src:libmojomojo-perl: "libmojomojo-perl: FTBFS: Tests failures"
  close, the underlying problem is fixed (pkg-perl)
• #832866 src:libclass-c3-xs-perl: "libclass-c3-xs-perl: FTBFS: Tests failures"
  upload new upstream release (pkg-perl)
• #834210 libdancer-plugin-database-core-perl: "libdancer-plugin-database-perl: FTBFS: Failed 1/5 test programs. 6/45 subtests failed."
  upload new upstream release (pkg-perl)
• #834793 libgit-wrapper-perl: "libgit-wrapper-perl: FTBFS: t/basic.t whitespace changes"
  add patch from upstream bug (pkg-perl)

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

• Presented "Reproducible Builds status update" at DebConf16, the annual Debian GNU/Linux developer conference.
• Attended a Core Infrastricture Initiative summit meeting. The CII was set up in the wake of the Heartbleed SSL vulnerability to support software projects that are critical to the functioning of the internet.

• Ensured that the Webconverger web kiosk operating system builds reproducibly. I may rework some of the patches to libisoburn and libisofs before sending them upstream. This work was sponsored by Webconverger.
• Proposed a pull request for Regex Replace (a Chrome extension to automatically replace text on webpages) to ensure that the rules were correctly HTML encoded on the options page. (#3)
• Proposed a change to ronn, a documentation generator that "is the opposite of roff", to make the output reproducible. (#98)
• Fixed an issue in django-enumfield, a custom Django web development field for type-safe named constants, to make the Enum.get interface more consistent. (#36)
• Proposed a change to txt2tags to make the output use SOURCE_DATE_EPOCH and non-timezone timestamps. (#204).

---

What happened in the Reproducible Builds effort between Sunday July 17 and Saturday July 23 2016: GSoC and Outreachy updates Valerie Young wrote an update about her Outreachy progress on tests.reproducible.org. Packages reviewed and fixed, and bugs filed Patches have been submitted by:

• Chris Lamb:
  • gearmand: please make the build reproducible
  • gngb: please make the build reproducible
  • speech-tools: please make the build reproducible
  • libwebsockets: please make the build reproducible
  • feh: please make the build reproducible
  • sa-exim: please make the build reproducible
  • wordgrinder: please make the build reproducible
  • ifmail: please make the build reproducible
  • libmongoc: please make the build reproducible
  • pxlib: please make the build reproducible
  • vtwm: please make the build reproducible
  • tcpreen: please make the build reproducible
  • borgbackup: please make the build reproducible
  • nfstrace: please make the build reproducible
  • ironic: please make the build reproducible
  • xloadimage: please make the build reproducible
  • stressapptest: please make the build reproducible
  • tcpflow: please make the build reproducible
  • python-mkdocs: please make the build reproducible
  • vtun: please make the build reproducible
  • hocr: please make the build reproducible
  • ui-auto: please make the build reproducible
  • ftplib: please make the build reproducible
  • fvwm: please make the build reproducible
  • why3: please make the build reproducible
  • uisp: please make the build reproducible
  • idjc: please make the build reproducible
  • python-docutils: please make the output of rst2man reproducible
  • python-mkdocs: please make the build reproducible
• Dhole:
  • libterralib: please make the build reproducible
• Reiner Herrmann:
  • aboot: please make the build reproducible

Package reviews 17 package reviews have been added and 4 have been updated. adding to our knowledge about identified issues. Some issues have been updated:

• Add new timestamps_in_manpages_generated_by_rst2man issue
• More timestamp issues are deterministic failures
• remove issue ftbfs_with_dpkg_buildpackage_buildinfo_identifier_error
• Add a patch for timestamps_in_manpages_generated_by_rst2man toolchain issue.

Weekly QA work FTBFS bugs have been reported by:

• Chris Lamb (5)
• Mattia Rizzolo (7)
• Samuel Hym (1)

diffoscope development

• Mattia Rizzolo:
  • Drop ar from RequiredToolNotFound.PROVIDERS as we don't use ar(1) directly
• Ximin Luo:
  • Add support for AR archives (including Rust .rlib files)
  • Add some more documentation for ArFile/ArContainer
  • RequiredToolNotFound.PROVIDERS dict values are package names, not binary names
• Chris Lamb:
  • Add git index file comparator.
  • Drop unused imports in Ar comparator.

strip-nondeterminism development

• strip-nondeterminism 0.020-1 was uploaded to unstable by Chris Lamb.
• Chris Lamb:
  • Correct the max number of bytes we read when copying PNG chunk data
  • Rewrite PNG handler to support bailing out on invalid header lengths
  • Include CRC when reading potential PNG blocks
  • Don't trust the value of $len in PNG handler. (Closes: #831443)
  • Improve readability in PNG handler by not including IEND detection in "else"

reprotest development

• Ceridwen:
  • Suppress autopkgtest verbosity for release
  • Add sdist and wheel to tests, and test the installed code
  • Maybe get setuptools install working
  • Merge virtualization into master
  • Reenable the Debian self-build
  • Ensure disorderfs is mounted with multiuser access if the container has root privileges
  • Rename Testbed.caps to Testbed.capabilities
  • Add Python license to debian/copyright
  • Fix cleanup handling iin both the failing and non-failing cases
  • Add new recommends and update version information
  • Add instructions for running the tests
  • Shorten license boilerplate and add new copyright info
  • Bump version numbers in preparation for the next release
  • Make the file names depend on the current version in the tests
  • Factor out common code for constructing shell ASTs
  • Move contextlib monkey-patch to its own file

tests.reproducible-builds.org

• Added main navigation highlighting to python pages (Valerie Young)
• Use python and templates to generate pkg sets (Valerie Young, with lots of review from mattia) which sped up runtime a lot, so package sets are now updated 4 times an hour instead of every 4 hours.

Misc. This week's edition was written by Chris Lamb and reviewed by a bunch of Reproducible builds folks on IRC.

The last few months have been a bit of a crazy period of ups and downs, with a tempest of events beneath the apparent and deceivingly calm surface waters of being unemployed (still at it). The daily grind Chief activities are, of course, those related to the daily grind of job-hunting, sending applications, and preparing and attending interviews. It is demoralising when one searches for many days or weeks without seeing anything suitable for one's skills or interests, or other more general life expectations. And it takes a lot of time and effort to put one's best in the applications for positions that one is really, really, interested in. And even for the ones which are meh, for a variety of reasons (e.g. one is not very suitable for what the offer demands). After that, not being invited to interviews (or doing very badly at them) is bad, of course, but quick and not very painful. A swift, merciful end to the process. But it's all the more draining when waiting for many weeks when not a few months with the uncertainty of not knowing if one is going to be lucky enough to be summoned for an interview; harbouring some hope one has to appear enthusiastic in the interviews, after all , while trying to keep it contained lest it grows too much ; then in the interview hearing good words and some praises, and feeling the impression that one will fit in, that one did nicely and that chances are good letting the hope grow again ; start to think about life changes that the job will require to make a quick decision should the offer finally arrives ; perhaps make some choices and compromises based on the uncertain result; then wait for a week or two after the interview to know the result... ... only to end up being unsuccessful. All the effort and hopes finally get squashed with a cold, short email or automatic response, or more often than not, complete radio silence from prospective employers, as an end to a multi-month-long process. An emotional roller coaster ^[1], which happened to me several times in the last few months. All in a day's work The months of preparing and waiting for a new job often imply an impasse that puts many other things that one cares about on hold, and one makes plans that will never come to pass. All in a day's (half-year's?) work of an unemployed poor soul. But not all is bad. This period was also a busy time doing some plans about life, mid- and long-term; the usual and some really unusual! family events; visits to and from friends, old and new; attending nice little local Debian gatherings or the bigger gathering of Debian SunCamp2016, and other work for side projects or for other events that will happen soon... And amidst all that, I managed to get some work done on aptitude. Two pictures worth (less than) a thousand bugs To be precise, worth 709 bugs 488 bugs in the first graph, plus 221 in the second. In 2015-11-15 (link to the post Work on aptitude): In 2016-06-18: Numbers The BTS numbers for aptitude right now are:

• 221 (259 if counting all merged bugs independently)
• 1 Release Critical (but it is an artificial bug to keep it from migrating to testing)
• 43 (55 unmerged) with severity Important or Normal
• 160 (182 unmerged) with severity Minor or Wishlist
• 17 (21 unmerged) marked as Forwarded or Pending

Highlights Beyond graphs and stats, I am specially happy about two achievements in the last year:

1. To have aptitude working today, first and foremost Apart from the abandon that suffered in previous years, I mean specifically the critical step of getting it through the troubles of the last summer, with the GCC-5/C++11 transition in parallel with a transition of the Boost library (explained in more detail in Work on aptitude). Without that, possibly aptitude would not have survived until today.
2. Improvements to the suggestions of the resolver In the version 0.8, there were a lot of changes related with improving the order of the suggestions from the resolver, when it finds conflicts or other problems with the planned actions. Historically, but specially in the last few years, there have been many complaints about the nonsensical or dangerous suggestions from the resolver. The first solution offered by the resolver was very often regarded as highly undesirable (for example, removal of many packages), and preferable solutions like upgrades of one or only a handful of packages being offered only after many removals; and keeps only offered as last resort.

Perhaps these changes don't get a lot of attention, given that in the first case it's just to keep working (with few people realising that it could have collapsed on the spot, if left unattended), and the second can probably go unnoticed because it just works or it started to work more smoothly doesn't get as much immediate attention as it suddenly broke! . Still, I wanted to mention them, because I am quite proud of those. Thanks Even if I put a lot of work on aptitude in the last year, the results of the graph and numbers have not been solely achieved by me. Special thanks go to Axel Beckert (abe / XTaran) and the apt team, David Kalnischkies and Julian Andres Klode who, despite the claim in that page, does not mostly work python-apt anymore... but also in the main tools. They help with fixing some of the issues directly, or changing things in apt that benefit aptitude, testing changes, triaging bugs or commenting on them, patiently explaining to me why something in libapt doesn't do what I think it does, and good company in general. Not the least, for holding impromptu BTS group therapy / support meetings, for those cases when prolonged exposure to BTS activity starts to induce very bad feelings. Thanks also to people who sent their translation updates, notified about corrections, sent or tested patches, submitted bugs, or tried to help in other ways. Change logs for details. Notes [1] ^{^} It's even an example in the Cambridge Dictionaries Online website, for the entry of roller coaster:

He was on an emotional roller coaster for a while when he lost his job.

This is a fusion recipe from a rather bland just stuff it with ricotta recipe I saw, David Scott s The Peniless Vegetarian , and my own mutations on those themes. I can t give you exact quantities, just make a little more than you will make the hollowed mound (grin), and the rest will make an excellent pasta sauce. Ingredients For an average sized butternut squash, you will need:
1 onion (I prefer red)
3 cloves of garlic
1 capsicum pepper (I prefer green, my ex- preferred red)
Some red lentils
Optional green or brown lentils for texture and flavour. I used some puy
The lentil quantity is hard to estimate, but I ratio 4 red to 1 optional.
Roughly one handful of chopped mushrooms i.e. when chopped, it is one handful
1 tin tinned tomatoes
Some tomato puree
A generous amout of garam masala garam masala is what brings out the flavout in lentils
Some paprike
Optional chilli if using chilli, I recommend fresh of course.
Optional Balsamic vinegar
Optional Marmite Preperation of the Squash
1. Cut the butternut squash in half, length ways. This is very hard, you will need a good large knife, and may require you jumping up and down into the air. This is the second most hard of the procedure. 2. For each half, scoop out the seeds, and pare back the bowl till it is no longer overly fibrous. Discard this, or find a use for the seeds. 3. For each half, scoop a channel of the softer flesh up from the baisin up near the top. This has to be done by feel, is hard and thankless work. Also experimentation required. Reserve this flesh. Preperation of the Filling This is just basically a nice lentil sauce that can be used with pasta, rice, toast etc. Important: this is not a stir fry, but a largish, heavy bottom pan is recommended. 1. Finely peel then chopp the onions and the garlic. Chopp the chillis if used (I am a chilli gal). Please observe Chilli Protocol[0] 2. Wash and chop the pepper and mushrooms. Not finely diced, but not crudite-sized slices. Remember that peppers shrivel down a little, mushrooms a lot. 3. Start frying the onions for a while in some oil (I prefer olive, but others are acceptible), until they just about to go translucent. Then add the garlic and optional chillis until the garlic is just cooking nicely. 4. Add the spices, turn over until all the containts of the pan are covered, and cook for another 30 seconds or so. Then add the tinned tomato, and then add half a can of cold water water which rinsed the tin out with. Stir this around, and make sure it is now at just at a simmer or pre-simmer. 5. Add the lentils. You want 0.5-1 cm of water above the lentils when you have added and stirred. Let these cook and expand for about 5 mins, stirring all the while, all the lentils will stick to the bottom. 6. Add the pepper, mushroom, reserved squash flesh, and optional dash of balsamic vinegar, and half a tea spoon of marmite. Cook and stir until the pepper goes soft. This is the hard part. Add boiling water if really too thick, or some tomato puree if too thin. There is no hard science to this, you want at the end of 10 minutes or so something resembling the thickness in texture of a stiff bolognaise sauce. Assembly
1. Have a baking tray. Whether you prefer to grease, line with foil, or line with baking parchment is up to you. I prefer baking parchment. 2. Stuff those two halves of butternut squash with that sauce you made. It should make a mound of about 1cm about the level. If you feel extravagent, and are not vegan, sprinkle a little grated cheese on top. 3. Place in a pre-heated oven of 200oC. Cooking time should be about 20 mins, but larger ones take longer. The acid test is to briefly take them out, and prod the lower side with a fork. It should go through the skin with little resistance. When ready, serve. It s really a dish in itself, but some people might like a bit of salad, or maybe a light green risotto.

I've been using oprofile for profiling programs for a while now (and especially QSoas, because it doesn't require specific compilation options, and doesn't make your program run much more slowly (like valgrind does, which can also be used to some extent for profiling). It's a pity the Debian package was dropped long ago, but the ubuntu packages work out of the box on Debian. But, today, while trying to see what takes so long in some fits I'm running, here's what I get:

~ operf QSoas
Unexpected error running operf: Permission denied

Looking further using strace, I could see that what was not working was the first call to perf_event_open.
It took me quite a long time to understand why it stopped working and how to get it working again, so here's for those of you who googled the error and couldn't find any answer (including me, who will probably have forgotten the anwser in a couple of months). The reason behing the change is that, for security reason, non-privileged users do not have the necessary privileges since Debian kernel 4.1.3-1; here's the relevant bit from the changelog:

  * security: Apply and enable GRKERNSEC_PERF_HARDEN feature from Grsecurity,
    disabling use of perf_event_open() by unprivileged users by default
    (sysctl: kernel.perf_event_paranoid)

The solution is simple, just run as root:

~ sysctl kernel.perf_event_paranoid=1

(the default value seems to be 3, for now). Hope it helps !

At LCA I attended a talk about Unikernels. Here are the reasons why I think that they are a bad idea: Single Address Space According to the Unikernel Wikipedia page [1] a significant criteria for a Unikernel system is that it has a single address space. This gives performance benefits as there is no need to change CPU memory mappings when making system calls. But the disadvantage is that any code in the application/kernel can access any other code directly. In a typical modern OS (Linux, BSD, Windows, etc) every application has a separate address space and there are separate memory regions for code and data. While an application can request the ability to modify it s own executable code in some situations (if the OS is configured to allow that) it won t happen by default. In MS-DOS and in a Unikernel system all code has read/write/execute access to all memory. MS-DOS was the least reliable OS that I ever used. It was unreliable because it performed tasks that were more complex than CP/M but had no memory protection so any bug in any code was likely to cause a system crash. The crash could be delayed by some time (EG corrupting data structures that are only rarely accessed) which would make it very difficult to fix. It would be possible to have a Unikernel system with non-modifyable executable areas and non-executable data areas and it is conceivable that a virtual machine system like Xen could enforce that. But that still wouldn t solve the problem of all code being able to write to all data. On a Linux system when an application writes to the wrong address there is a reasonable probability that it will not have write access and you will immediately get a SEGV which is logged and informs the sysadmin of the address of the crash. When Linux applications have bugs that are difficult to diagnose (EG buffer overruns that happen in production and can t be reproduced in a test environment) there are a variety of ways of debugging them. Tools such as Valgrind can analyse memory access and tell the developers which code had a bug and what the bug does. It s theoretically possible to link something like Valgrind into a Unikernel, but the lack of multiple processes would make it difficult to manage. Debugging A full Unix environment has a rich array of debugging tools, strace, ltrace, gdb, valgrind and more. If there are performance problems then tools like sysstat, sar, iostat, top, iotop, and more. I don t know which of those tools I might need to debug problems at some future time. I don t think that any Internet facing service can be expected to be reliable enough that it will never need any sort of debugging. Service Complexity It s very rare for a server to have only a single process performing the essential tasks. It s not uncommon to have a web server running CGI-BIN scripts or calling shell scripts from PHP code as part of the essential service. Also many Unix daemons are not written to run as a single process, at least threading is required and many daemons require multiple processes. It s also very common for the design of a daemon to rely on a cron job to clean up temporary files etc. It is possible to build the functionality of cron into a Unikernel, but that means more potential bugs and more time spent not actually developing the core application. One could argue that there are design benefits to writing simple servers that don t require multiple programs. But most programmers aren t used to doing that and in many cases it would result in a less efficient result. One can also argue that a Finite State Machine design is the best way to deal with many problems that are usually solved by multi-threading or multiple processes. But most programmers are better at writing threaded code so forcing programmers to use a FSM design doesn t seem like a good idea for security. Management The typical server programs rely on cron jobs to rotate log files and monitoring software to inspect the state of the system for the purposes of graphing performance and flagging potential problems. It would be possible to compile the functionality of something like the Nagios NRPE into a Unikernel if you want to have your monitoring code running in the kernel. I ve seen something very similar implemented in the past, the CA Unicenter monitoring system on Solaris used to have a kernel module for monitoring (I don t know why). My experience was that Unicenter caused many kernel panics and more downtime than all other problems combined. It would not be difficult to write better code than the typical CA employee, but writing code that is good enough to have a monitoring system running in the kernel on a single-threaded system is asking a lot. One of the claimed benefits of a Unikernel was that it s supposedly risky to allow ssh access. The recent ssh security issue was an attack against the ssh client if it connected to a hostile server. If you had a ssh server only accepting connections from management workstations (a reasonably common configuration for running servers) and only allowed the ssh clients to connect to servers related to work (an uncommon configuration that s not difficult to implement) then there wouldn t be any problems in this regard. I think that I m a good programmer, but I don t think that I can write server code that s likely to be more secure than sshd. On Designing It Yourself One thing that everyone who has any experience in security has witnessed is that people who design their own encryption inevitably do it badly. The people who are experts in cryptology don t design their own custom algorithm because they know that encryption algorithms need significant review before they can be trusted. The people who know how to do it well know that they can t do it well on their own. The people who know little just go ahead and do it. I think that the same thing applies to operating systems. I ve contributed a few patches to the Linux kernel and spent a lot of time working on SE Linux (including maintaining out of tree kernel patches) and know how hard it is to do it properly. Even though I m a good programmer I know better than to think I could just build my own kernel and expect it to be secure. I think that the Unikernel people haven t learned this.

• [1] https://en.wikipedia.org/wiki/Unikernel

Yesterday we have released CafeOBJ 1.5.5 with a long list of changes, and many more internal changes. Documentation pages have been updated with the latest reference manual (PDF, Html) as well as some new docs on CITP (in Japanese for now) and tutorials. To quote from our README:

CafeOBJ is a new generation algebraic specification and programming language. As a direct successor of OBJ, it inherits all its features (flexible mix-fix syntax, powerful typing system with sub-types, and sophisticated module composition system featuring various kinds of imports, parameterised modules, views for instantiating the parameters, module expressions, etc.) but it also implements new paradigms such as rewriting logic and hidden algebra, as well as their combination.

Changes

• Enhancemets of a family of Bool term inspector commands:
  • binspect [ in : <Modexpr> ] <bool-term> . converts given <bool-term> into abstracted xor-and normal form
  • bshow [ tree grind ] shows abstracted Bool term
  • bresolve [ <num1> [ <num2> all ] ] shows assignments which make abstracted Bool term true
  • bguess imply and or tries with some heuristics to solve the abstracted Bool term
• bug fix in ACZ rewriting
• CITP changes
  • :ctf [ <term> ] accepts constructors with arguments
  • new command :imply to make instantiated lhs of existing equation of the form eq lhs = true a premise of a goal sentence
  • new tactic rd- which is similar to rd but cancels the normalization of the goal sentence if the sentence is not satisfied
• new switch tree horizontal. if on show term tree displays the term tree structure horizontally (default off).

Availability Binary packages for Windows, MacOS, and Linux have been built, and can be found at the CafeOBJ download page. The source code can also be found on the download page, or directly from here: cafeobj-1.5.5.tar.gz. The CafeOBJ Debian package will be updated soon. Macports file has also been updated, please see the above download/install page for details how to add our sources to your macport. Bug reports If you find a bug, have suggestions, or complains, please open an issue at the issue tracker for CafeOBJ. For other inquiries, please use info@cafeobj.org

Now I just have to finish putting all the systems in there and I will be able to call the project a success.

Writing programs is fun, but making them fast can be a pain. Python programs are no exception to that, but the basic profiling toolchain is actually not that complicated to use. Here, I would like to show you how you can quickly profile and analyze your Python code to find what part of the code you should optimize. What's profiling? Profiling a Python program is doing a dynamic analysis that measures the execution time of the program and everything that compose it. That means measuring the time spent in each of its functions. This will give you data about where your program is spending time, and what area might be worth optimizing. It's a very interesting exercise. Many people focus on local optimizations, such as determining e.g. which of the Python functions range or xrange is going to be faster. It turns out that knowing which one is faster may never be an issue in your program, and that the time gained by one of the functions above might not be worth the time you spend researching that, or arguing about it with your colleague. Trying to blindly optimize a program without measuring where it is actually spending its time is a useless exercise. Following your guts alone is not always sufficient. There are many types of profiling, as there are many things you can measure. In this exercise, we'll focus on CPU utilization profiling, meaning the time spent by each function executing instructions. Obviously, we could do many more kind of profiling and optimizations, such as memory profiling which would measure the memory used by each piece of code something I talk about in The Hacker's Guide to Python. cProfile Since Python 2.5, Python provides a C module called cProfile which has a reasonable overhead and offers a good enough feature set. The basic usage goes down to:
Though you can also run a script with it, which turns out to be handy:
This prints out all the function called, with the time spend in each and the number of times they have been called. Advanced visualization with KCacheGrind While being useful, the output format is very basic and does not make easy to grab knowledge for complete programs. For more advanced visualization, I leverage KCacheGrind. If you did any C programming and profiling these last years, you may have used it as it is primarily designed as front-end for Valgrind generated call-graphs. In order to use, you need to generate a cProfile result file, then convert it to KCacheGrind format. To do that, I use pyprof2calltree.
And the KCacheGrind window magically appears!

Concrete case: Carbonara optimization I was curious about the performances of Carbonara, the small timeserie library I wrote for Gnocchi. I decided to do some basic profiling to see if there was any obvious optimization to do. In order to profile a program, you need to run it. But running the whole program in profiling mode can generate a lot of data that you don't care about, and adds noise to what you're trying to understand. Since Gnocchi has thousands of unit tests and a few for Carbonara itself, I decided to profile the code used by these unit tests, as it's a good reflection of basic features of the library. Note that this is a good strategy for a curious and naive first-pass profiling. There's no way that you can make sure that the hotspots you will see in the unit tests are the actual hotspots you will encounter in production. Therefore, a profiling in conditions and with a scenario that mimics what's seen in production is often a necessity if you need to push your program optimization further and want to achieve perceivable and valuable gain. I activated cProfile using the method described above, creating a cProfile.Profile object around my tests (I actually started to implement that in testtools). I then run KCacheGrind as described above. Using KCacheGrind, I generated the following figures.

The test I profiled here is called test_fetch and is pretty easy to understand: it puts data in a timeserie object, and then fetch the aggregated result. The above list shows that 88 % of the ticks are spent in set_values (44 ticks over 50). This function is used to insert values into the timeserie, not to fetch the values. That means that it's really slow to insert data, and pretty fast to actually retrieve them. Reading the rest of the list indicates that several functions share the rest of the ticks, update, _first_block_timestamp, _truncate, _resample, etc. Some of the functions in the list are not part of Carbonara, so there's no point in looking to optimize them. The only thing that can be optimized is, sometimes, the number of times they're called.

The call graph gives me a bit more insight about what's going on here. Using my knowledge about how Carbonara works, I don't think that the whole stack on the left for _first_block_timestamp makes much sense. This function is supposed to find the first timestamp for an aggregate, e.g. with a timestamp of 13:34:45 and a period of 5 minutes, the function should return 13:30:00. The way it works currently is by calling the resample function from Pandas on a timeserie with only one element, but that seems to be very slow. Indeed, currently this function represents 25 % of the time spent by set_values (11 ticks on 44). Fortunately, I recently added a small function called _round_timestamp that does exactly what _first_block_timestamp needs that without calling any Pandas function, so no resample. So I ended up rewriting that function this way:
And then I re-run the exact same test to compare the output of cProfile.

The list of function seems quite different this time. The number of time spend used by set_values dropped from 88 % to 71 %.

The call stack for set_values shows that pretty well: we can't even see the _first_block_timestamp function as it is so fast that it totally disappeared from the display. It's now being considered insignificant by the profiler. So we just speed up the whole insertion process of values into Carbonara by a nice 25 % in a few minutes. Not that bad for a first naive pass, right?

The Linux Kernel Oops website collects kernel errors from all over the World helping kernel developers finding issues occurring in the wild but they cannot help if no one sends reports to them. The Kerneloops client used to be part of Debian releases but it has been removed from the archive due to not working with the new collector site. When I started observing oopses on my machine I first thought of submitting a bug against the linux package in BTS, but looking at the numerous bugs opened already I looked for a more automated solution which would also help others. Reviving the kerneloops package involved switching it to the new submission URL, fixing a few memory allocation bugs in C (this is the first package I found using Valgrind by default for running tests) and ensuring that upstream was still active. The last step took the most of the time but finally Anton Arapov kindly accepted my patches and everything was set for the new upload. The package is now available from unstable and if you feel so (especially if you experience oopses) please give it a try and report any problems you find. I m also happy to receive success stories about oopses fixed after discovering and collecting them with the client.

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 8 hours on Debian LTS. In that time, I mostly did CVE triaging (in the last 3 days since I m of LTS frontdesk duty this week). I pushed 14 commits to the security tracker. There were multiple CVE without any initial investigation so I checked the status of the CVE not only in squeeze but also in wheezy/jessie. On unpaid time, I wrote and sent the summary of the work session held during DebConf. And I tried to initiate a discussion about offering mysql-5.5 in squeeze-lts. We also have setup lts-security@debian.org so that we can better handle embargoed security updates. The Debian Administrator s Handbook I spent a lot of time on my book, the content update has been done but now we re reviewing it before preparing the paperback. I also started updating its French translation. You can help review it too. While working on the book I noticed that snort got removed from jessie and the SE linux reference policy as well. I mailed their maintainers to recommend that they provide them in jessie-backports at least those packages are relatively important/popular and it s a pity that they are missing in jessie. I hope to finish the book update in the next two weeks! Distro Tracker I spent a lot of time to revamp the mail part of Distro Tracker. But as it s not finished yet, I don t have anything to show yet. That said I pushed an important fix concerning the mail subscriptions (see #798555), basically all subscriptions of packages containing a dash were broken. It just shows that the new tracker is not yet widely used for mail subscription I also merged a patch from Andrew Starr-Bochicchio (#797633) to improve the description of the WNPP action items. And I reviewed another patch submitted by Orestis Ioannou to allow browsing of old news (see #756766). And I filed #798011 against bugs.debian.org to request that a new X-Debian-PR-Severity header field be added to outgoing BTS mail so that Distro Tracker can filter mails by severity and offer people to subscribe to RC bugs only. Misc Debian work I filed many bugs this month and almost all of them are related to my Kali work:

• 3 on debootstrap: #798560 (request for suite-config option), #798562 (allow sharing bootstrap scripts), #7985604 (request to add kali related bootstrap scripts).
• 3 requests of new upstream versions: for gpsd (#797899), for valgrind (#800013) and for puppet (#798636).
• #797783: sbuild fails without any error message when /var/lib/sbuild is not writable in the chroot
• #798181: gnuradio: Some files take way too long to compile (I had to request a give-back on another build daemon to ensure gnuradio migrated back to testing, and Julien Cristau suggested that it would be better to fix the package so that a single file doesn t take more than 5 hours to build )
• #799550: libuhd003v5 lost its v5 suffix

Thanks See you next month for a new summary of my activities.

3 comments Liked this article? Click here. My blog is Flattr-enabled.

What happened in the reproducible builds effort this week: Toolchain fixes

• Joachim Breitner uploaded haskell-devscripts/0.9.11 which ads support for UTF-8 encoded debian/control file with all locales. Original patch by Chris Lamb.
• Jonas Smedegaard uploaded ghostscript which adds support for SOURCE_DATE_EPOCH. Original patch by Dhole.

akira submitted a patch to make cdbs export SOURCE_DATE_EPOCH. She uploded a package with the enhancement to the experimental reproducible repository. Packages fixed The following 15 packages became reproducible due to changes in their build dependencies: dracut, editorconfig-core, elasticsearch, fish, libftdi1, liblouisxml, mk-configure, nanoc, octave-bim, octave-data-smoothing, octave-financial, octave-ga, octave-missing-functions, octave-secs1d, octave-splines, valgrind. The following packages became reproducible after getting fixed:

• ant-contrib/1.0~b3+svn177-7 by Emmanuel Bourg.
• authbind/2.1.1+nmu1 uploaded by Johannes Schauer, original patch by akira.
• elektra/0.8.12-1 uploaded by Pino Toscano, original patch by akira.
• glance/2015.1.0-3 by Thomas Goirand.
• gnumeric/1.12.23-1 uploaded by Dmitry Smirnov, original patch by Daniel Kahn Gillmor.
• libdevice-cdio-perl/0.3.0-5 uploaded by gregor herrmann, report by Chris Lamb.
• libxray-scattering-perl/3.0.1-1.1 uploaded by gregor herrmann, original patch by Reiner Herrmann.
• libxray-spacegroup-perl/0.1.1-2.1 uploaded by gregor herrmann, original patch by Chris Lamb.
• ryu/3.23-2 by Dariusz Dwornikowski.

Some uploads fixed some reproducibility issues but not all of them:

• apophenia/0.999e+ds-1 uploaded by Jerome Benoit, original patch by akira.
• xbs/0-10 uploaded by Matthew Vernon, original patch by Colin Watson.
• bbswitch/0.8-2 uploaded by Vincent Cheng, original patch by Chris Lamb.
• mariadb-10.0/10.0.20-3 by Otto Kek l inen.
• icedove/38.0.1-1 uploaded by Christoph Goehre, improvements by Carsten Schoenert.
• apache2/2.4.16-1 uploaded by Stefan Fritsch, improvements by Jean-Michel Vourg re.

In contrib, Dmitry Smirnov improved libdvd-pkg with 1.3.99-1-1. Patches submitted which have not made their way to the archive yet:

• #793705 on mime-support by akira: set the mtimes of all files which are modified during builds to the latest debian/changelog entry.
• #793922 on polymake by Chris Lamb: sort Perl hash keys.
• #793980 on lsof by Valentin Lorentz: removes extra informations from the build system..
• #793996 on at by Valentin Lorentz: use absolute values for chmod.
• #794005 on python-dtcwt by Chris Lamb: use a constant seed for the random number generator.
• #794011 on gzip by Valentin Lorentz: remove date from texinfo header.
• #794014 on moin by Dhole: normalizes the timezone and fixes timestamps from the files compressed with zip.
• #794106 on cryptsetup by Valentin Lorentz: use date from latest debian/changelog entry in manpage.
• #794130 on coq by Valentin Lorentz: use C locale and UTC timezone when generating the build date.
• #794225 on libsyncml by akira: export SOURCE_DATE_EPOCH.
• #794239 on zipios++ by akira: export SOURCE_DATE_EPOCH.
• #794247 on whizzytex by Dhole: remove current date from documentation.
• #794248 on cortado by Dhole: allow the build date to be set externally and use time from the latest debian/changelog entry.
• #794395 on classified-ads by Reiner Herrmann: remove timestamps from PNG images.
• #794398 on clhep by Reiner Herrmann: sort source file list.
• #794399 on parsec47 by Reiner Herrmann: use C locale when sorting source files.
• #794400 on tumiki-fighters by Reiner Herrmann: use C locale when sorting source files.

reproducible.debian.net Four armhf build hosts were provided by Vagrant Cascadian and have been configured to be used by jenkins.debian.net. Work on including armhf builds in the reproducible.debian.net webpages has begun. So far the repository comparison page just shows us which armhf binary packages are currently missing in our repo. (h01ger) The scheduler has been changed to re-schedule more packages from stretch than sid, as the gcc5 transition has started This mostly affects build log age. (h01ger) A new depwait status has been introduced for packages which can't be built because of missing build dependencies. (Mattia Rizzolo) debbindiff development Finally, on August 31st, Lunar released debbindiff 27 containing a complete overhaul of the code for the comparison stage. The new architecture is more versatile and extensible while minimizing code duplication. libarchive is now used to handle cpio archives and iso9660 images through the newly packaged python-libarchive-c. This should also help support a couple other archive formats in the future. Symlinks and devices are now properly compared. Text files are compared as Unicode after being decoded, and encoding differences are reported. Support for Sqlite3 and Mono/.NET executables has been added. Thanks to Valentin Lorentz, the test suite should now run on more systems. A small defiency in unquashfs has been identified in the process. A long standing optimization is now performed on Debian package: based on the content of the md5sums control file, we skip comparing files with matching hashes. This makes debbindiff usable on packages with many files. Fuzzy-matching is now performed for files in the same container (like a tarball) to handle renames. Also, for Debian .changes, listed files are now compared without looking the embedded version number. This makes debbindiff a lot more useful when comparing different versions of the same package. Based on the rearchitecturing work has been done to allow parallel processing. The branch now seems to work most of the time. More test needs to be done before it can be merged. The current fuzzy-matching algorithm, ssdeep, has showed disappointing results. One important use case is being able to properly compare debug symbols. Their path is made using the Build ID. As this identifier is made with a checksum of the binary content, finding things like CPP macros is much easier when a diff of the debug symbols is available. Good news is that TLSH, another fuzzy-matching algorithm, has been tested with much better results. A package is waiting in NEW and the code is ready for it to become available. A follow-up release 28 was made on August 2nd fixing content label used for gzip2, bzip2 and xz files and an error on text files only differing in their encoding. It also contains a small code improvement on how comments on Difference object are handled. This is the last release name debbindiff. A new name has been chosen to better reflect that it is not a Debian specific tool. Stay tuned! Documentation update Valentin Lorentz updated the patch submission template to suggest to write the kind of issue in the bug subject. Small progress have been made on the Reproducible Builds HOWTO while preparing the related CCCamp15 talk. Package reviews 235 obsolete reviews have been removed, 47 added and 113 updated this week. 42 reports for packages failing to build from source have been made by Chris West (Faux). New issue added this week: haskell_devscripts_locale_substvars. Misc. Valentin Lorentz wrote a script to report packages tested as unreproducible installed on a system. We encourage everyone to run it on their systems and give feedback!

This was my fifth month working on Debian LTS. I was assigned 16 hours by Freexian's Debian LTS initiative. I worked on several packages but haven't uploaded updates yet. linux-2.6 There is a steady stream of security issues in all Linux kernel versions. As we don't want to prompt reboots too often, these won't necessarily result in an update every month; it depends on the severity of the issue(s). However, I triaged the current issues and committed the available fixes to version control. dpkg The PGP signature validation in dpkg-source has a number of bugs, including CVE-2015-0840 which allows the validation to be completely subverted. I backported fixes for these from the 1.16 (wheezy) branch to the 1.15 (squeeze) branch. dpkg has a test suite and the fixes all came with new regression tests, so this was easy to verify. As dpkg is a native package (i.e. there is little separation between the 'upstream' and Debian packaging), I preferred that the dpkg maintainers would turn this into an 'upstream' release that I could upload. Guillem Jover has now reviewed and (semi-)released my changes, so I will complete this update soon. tiff A large number of bugs have been found in the venerable tiff library and tools, mostly by 'fuzzing' them with afl. The bugs have been unhelpfully grouped into a smaller number of CVE IDs, although it's not clear that these groups were introduced at the same time and they certainly weren't fixed at the same time. Most of the upstream bug reports come with samples to reproduce the bug (crash or memory corruption detectable with valgrind), but many of these did not turn out to be reproducible (either upstream or with the version in 'squeeze'). Where they were reproducible, I've verified that the patches fix the issue. Unfortunately tiff does not have a test suite, so I made a call for testing on the debian-lts list. If I don't get any responses soon, I'll run my own basic tests with programs that use libtiff before uploading.

As a follow-up to my post about bootstrapping arm64 in Debian, we've had more hardware given to Debian for us to use in porting and building packages for arm64. Applied Micro sent me an X-Gene development machine to set up and use. Unfortunately, the timing was unlucky and the machine sat on my desk unopened for a few weeks while I was on long holiday in Australia. Once I was back, I connected it up and got it working. Out of the box, a standard Jessie arm64 installation worked using network boot (dhcp and tftp). I ran through d-i as normal and installed a working system, then handed it over to the DSA and buildd folks to get the machine integrated into our systems. Easy! The machine is now up and running as arm-arm-03.debian.org and has been building packages for a few weeks now. You can see the stats here on the buildd.debian.org site. In terms of installation, I also got the machine to boot using one of our netinst images on a USB stick, but that path didn't get very far. The USB drivers for this hardware have only quite recently gone into the mainline kernel, and haven't been backported to the Debian Jessie kernel yet. I'm hoping to get those included shortly. There's also an option to replace the U-Boot firmware that came with the X-Gene with UEFI instead, which would be much more helpful for a server platform like this. I'll look into doing that upgrade soon too, but probably after the Jessie release is done. I don't want to jinx things just now. *grin* Thanks to APM for their generous donation here, and particularly to Richard Zenkert for his help in getting this machine shipped to us.